Just gotta restrain yourself when you're around non Z'ers. You can probably understand how it gets annoying to them- like when one of your friends is obsessed with a girl and keeps talking about her; you don't give a shit and he won't shut up. We've all been there. Now the situation is just reversed, your friends don't really give a shit about the Z (at least aren't obsessive with it), and if you keep talking about it, even though you don't realize it, you'll annoy them probably.
Just keep the Z seperate from your normal social life =)